Archived source — recreation of
Seung lab proofreading-annotation empire operation manual (szichieh, 2024). ·
Open the live original ↗ · Retrieved 2026-05-26. Point-in-time snapshot.
Note: some documents were mentioned multiple times across different sections. Consistent naming was used throughout for easier search and reference.
Human Data HQ_Operation Manual
PU
- AN/DN matching + proofreading (Austin, James, Kyle, Ryan, Ben)
- (Jay) interneuron BANC-MANC matching
- Doug's current tasks include:
- Checking cells with the “ripped area” status in the BANC mastersheet (tab: 12.v305_OrphanBB_VNC500+syn) and leaving notes when it’s okay to mark a cell as complete.
- Periodically checking the #help_proofreading channel for any issues that need attention.
- Doug has been incredibly helpful in reviewing some of 611’s work and providing guidance. He could also serve as an additional manpower resource.
- Jay’s ToDo: 611 daily stats from CA3 trainees (611 daily stats_training - CA3 1 cohort)
- Until a replacement for Szi-chieh is found, PU tracers should submit their time-off requests to May.
611 - fly
BANC project:
Complete:
Master sheet - BANC
- AN and DN
- segments with soma in VNC
- segments with soma in CB
- "deorphenize" soma in VNC+CB (note: Some segments remained that were difficult to deorphanize.)
- proofread orphan backbones from upstream/downstream cells of AN and DN
- partial OL
- attaching orphan backbones with 500+ synapses in VNC
- Current/Future priority (major team):
Note: Segments in the VNC take priority for proofreading over those in the CB, unless otherwise directed by Mala or researchers from Wei’s lab.
For reference: FAFB proofreading strategy -> segments have either more than 100 synapses or at least one connection with ≥ 15
- attaching orphan backbones with 500+ synapses in the CB
- Trace from seeded neurons in the nerves, this would help to complete proofreading of sensory/motor neurons
- Attach orphan backbones with medium synapse threshold (eg. 500-200 synapses)
- Review neurons that have poor NBLAST scores, as this is indicative of tracing error (Note: 611 haven’t done any matching using NBLAST before, additional training may be needed)
- Fix priority neurons
- Current/Future priority (5 agents: Marion Roda, James Nicle Cano, Michael Mates, Lea Lovely Joy Pelingon, Philip Lenard Ampo):
Sheet - AN/DN matching
- Focus on the ones in the list that don’t have a good pair and only do proofreading (rows with “both comparison_root_id_latest” and “comparison_root_id” are 0)
- Based on the work of the current five agents and the number of outstanding cells to proofread in the sheet, please decide whether additional agents are needed.
611 - Minnie
Master sheet - Minnie
Note1: Two agents from the CA3 new hires will:
- Once complete their CA3 training.
- They will continue practicing in the Minnie sandbox.
- Once their proofreading skills are deemed sufficient in Minnie sandbox, they will transition to Minnie Production. (We’ll use cells proofread by the current team for training and score calculations)
Note2: The Minnie team and the CA3 new hires have slightly different shifts. When the two CA3 new hires begin working in the Minnie sandbox, we can ask if they prefer to switch to the Minnie shift or stay on the CA3 new hire shift. Since the CA3 new hire TL and OIC are familiar with Minnie proofreading, they can continue overseeing these two agents if the agents choose to remain on their original shift.
611 - CA3 (Production: fly agents)
Master sheet - CA3
Complete:
PyC proofreading
Tab: “1.pyramidal_cell_list”
- proofreading Pyramidal-Thorny-Extensive axon
- proofreading Pyramidal-Little Thorny-Extensive axon
- proofreading Pyramidal-Athorny-Extensive axon
Mossy Fiber proofreading
- Tab: “4.MFunique_611#1” & “5.MFtoKnownThorny_611#2”
PyC proofreading
Tab: “6.Additional_pyramidal”
- proofreading Pyramidal-Thorny-Extensive axon
- proofreading Pyramidal-Little Thorny-Extensive axon
- proofreading Pyramidal-Athorny-Extensive axon
Current/Future priority:
- proofreading Pyramidal-Thorny or Athorny-Minimal axon (Tab: “1.pyramidal_cell_list”)
- proofreading Pyramidal-Thorny or Athorny-Minimal axon (Tab: “6.Additional_pyramidal”)
- Proofreading Inh (no soma; Tab: “10.inhibitory_axons”)
- proofreading Inh-Extensive axon (Tab: “1.pyramidal_cell_list”)
- proofreading Inh-Minimal axon (Tab: “1.pyramidal_cell_list”)
- proofreading Inh-Extensive axon (Tab: “6.Additional_pyramidal”)
- proofreading Inh-Minimal axon (Tab: “6.Additional_pyramidal”)
611 - CA3 New Hire
- New hire start date: Sep 2, 2024 (9/2/2024-9/5/2024: practice random cells in Minnie. 9/5/2024: Start working in Pinky sandbox)
- Training ToDo
- To be decided: The score threshold for cells to be considered good quality after proofreading. We can base this on the following sets:
- Pinky sandbox (layer: pinky_training) proofread by expert proofreaders (fly agents)
- Pinky sandbox (layer: pinky_training1) proofread by new hire
- For reference, FW-FAFB threshold is 96%
Complete:
- Pinky sandbox (layer: pinky_training1)
Current/Future priority:
- Pinky sandbox (layer: pinky_training2)
- Pinky sandbox (layer: pinky_training3) => should ask Akhilesh to create this set.
- Admit trainee based on their proofreading ability (Production admission checklist - mouse)
- Add trainees who get into Prod. in the 611 daily stats - mouse sheet
- The PI/Lab Head added in the admission panel): Sebastian Seung Lab
- Join the proofreading priority with 611 - CA3 (Production: fly agents) team.
Common gmail (G drive) for proofreading/annotating team
List of 611 managing personnel (TL + OIC)
Operation Manager: Nats Feria (team leads’ supervisor)
Email: rferia@sixelevencenter.com
Slack: https://flywire-forum.slack.com/team/U05FS3R2MCH
Team Lead & OIC:
TL: @Queendolyn Pama @Jet @John Raphael Toledo @Michelle Pantujan
OIC: @Joshua Bautista Bañez @Ramor @mark lloyd @Nelskie @Raffy Acosta
TL: @Mai Dumale
OIC: @AL
TL: @Rei
OIC: @El
- Current TL & OIC from mouse datasets (CA3 and Minnie) join each other’s meeting
- For agents: Minnie team joins the meeting with CA3 (production and new hire). CA3 teams don’t join the Minnie team meeting
General info about 611 training
- During training, agents receive scores for individual cells they proofread. We refrain from providing answers unless they have attempted multiple times and are still unable to pinpoint the errors. This approach helps them learn to identify potential issues independently, ensuring they do not simply follow instructions without understanding.
- Here is the script (FW training score analysis) for monitoring the weekly improvement of trainees. The data primarily used include, but are not limited to:
- Weekly average score for each individual
- Weekly number of cells completed by each individual
- Overall average score for each individual
- Total number of cells completed by each individual
- Box plot from all cells for each individual
- Calculation of productivity
- Overall performance during training, encompassing both productivity and accuracy
- Proofreader percentile chart
- Visualization of individual progress, both overall and weekly, focusing on accuracy
=> I provide some of the above data to TL and OIC on a weekly basis to tailor proofreading training plans. Additionally, we utilize this data along with insights from TL and OIC to manage a rolling base admission into Production, as detailed in the “Production Admission Checklist.”
Other Relevant Documents:
Production admission checklist - fly
Production admission checklist - mouse
611 daily stats - fly
611 daily stats - mouse
611 Trainee Candidate_EW pre-screening scores (both mouse and fly)
CA3 training related:
Who should you reach?
- Production Admission: Marissa
- Please provide emails to Marissa
- Marissa Pratt: Please add PI/Lab Head info and desired group when admitting into Production.
Currently I have:
- “Mala Murthy Lab, Sebastian Seung Lab” for any agent working in fly (FAFB, BANC)
- “Sebastian Seung Lab” for any agent working in mouse (Minnie, CA3)
- EM Compression and Asia Storage: Will
- For every new dataset that 611 works on, we compress the EM and store its copy in Asia to help the agents load the dataset.
- We need to inform Will about the compression quality we want to use. A typical factor is q=85. We could ask Will to run a few tests to help us decide on the best factor.
Here are the previous setups for reference:
- Pinky: q (compression quality) =75
- CA3: q=85
- Minnie: q=90
- BANC: q=75
- Make a sandbox/training copy: Akhilesh
- If something is urgent and you don't hear from him on any channels, just send him a direct message and he will check.
- General practice: when the dataset is huge (e.g. Minnie), we revert the edits. When the dataset is small (e.g. Pinky), we make a copy directly.
- Stats & Training score calculation: Jay
Current Meetings:
* Note: The Philippines doesn’t have daylight saving time. The time listed below is from daylight saving. It will change in the winter when changed to standard time.
- Meeting with 611 operations manager, Nats (weekly): 10:30a EDT every Wednesday
Individual meeting:
- Minnie (biweekly) with Bethany
Group meetings for each team:
- CA3 (Production; fly agents; weekly): 8:10p EDT every Monday
- Fly (weekly): 8:10p EDT every Wednesday
- CA3 (New Hire; weekly): 5:05p EDT every Thursday
- Minnie (biweekly): 8:10p EDT every other Thursday
Current meeting content:
- CA3 (Production; fly agents; weekly):
- Going over current task and answer questions
- Training or intro for new proofreading task
- Fly (weekly): 8:10p EDT every Wednesday
- Going over “need 2nd opinion” tag from the sheet or unanswered questions from #help_proofreading channel
- Answer questions that occur to agents during proofreading
- CA3 (New Hire; weekly): 5:05p EDT every Thursday
- Proofreading training curriculum
- Common errors from proofread cells
- Minnie (biweekly): 8:10p EDT every other Thursday
- Problem of the week: submitted by the agents and/or selected by Celia
Miscellaneous:
- Please monitor #help_proofreading & #help_proofreading_minnie channels for proofreading questions
- We use #611_ca3-proofreading temporarily for proofreading questions.
- WFH requests from agents should be sent to the person who is responsible for the specific team and May
- When granting WFH, please ensure the TL verifies that the employee's home internet connection and equipment are suitable.
- Please record the date, details including the length of WFH requests in the "Logs" tab of the relevant daily stats sheet.
About the Future
(This was Szi-chieh’s little wonderland. 😂 )
Objective: Transform the 611 new proofreader training into a more systematic and replicable process.
- Refine Current Training Checklist: Continuously refine training checklist to ensure all essential tasks and procedures for preparing, executing, and following up with new training cohorts stay aligned with evolving needs. For example, here is one from the CA3 training (Proofreading training checklist)
- Refine Proofreading Teaching Curriculum: CA3 training curriculum example
- Refine Teaching Documentation
- Design Practical Training Exercises: Develop specific practice examples for trainees that align with the teaching topics, enhancing hands-on learning and application of concepts.
- Automate Performance Monitoring: Aim to automate the generation of weekly performance statistics and graphs.
- Training Analysis Shared with Team Lead: This includes individual cell completion rates, average scores and weekly improvement to spot any declines in proofreading and enable early intervention with additional training. Here’s an example of the data Szi-chieh shares with the TL and OIC weekly to help customize proofreading training and make decisions on Production admissions. Currently, the script (FW training score analysis) is run manually. We could automate it, so the script runs on its own and the plots and analysis are automatically added to the slide deck above.
- For Monitoring Performance:
- Create dashboard to monitor proofreader performance and project progress
- We can start from NeuVue dashboard and expand the data from there.
- NeuVue development:
- NeuVue could be a great tool for managing proofreading tasks, allowing us to move away from Google Sheets, where at some point, we hit a limit on the number of simultaneous users.
- We may do progress tracking and productivity monitoring in NeuVue, too. Casey (from Allen Institute) is working on integrating Spelunker into the NeuVue interface, and Danny (from APL) secured funding to support task management and add new features for proofreading tasks. We can collaborate with them.
- Automate Proofreader Training: Explore the possibilities using LLM for proofreader training to automate the training process. (I think this project would be really fun and beneficial!)